{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation \n",
    "\n",
    "`advsecurenet` supports both benign and adversarial evaluation. In benign evaluation, the model is evaluated on clean images. In adversarial evaluation, there are multiple options to evaluate the model, the defense or the attack itself. Currently supported adversarial evaluations are: \n",
    "\n",
    "1. **Attack Success Rate**: The ratio of successful adversarial examples generated by the attack. Successful adversarial examples are those that are initially correctly classified by the model and are then misclassified after the attack. If the attack is untargeted, the misclassification should be to any class other than the original class. If the attack is targeted, the misclassification should be to the target class.\n",
    "\n",
    "2. **Perturbation Distance**: The average perturbation distance between the original and adversarial examples. The perturbation distance is calculated as the Lp norm of the difference between the original and adversarial examples.\n",
    "\n",
    "3. **Transferability**: The ratio of successful adversarial examples that fool both the model and a surrogate model.\n",
    "\n",
    "4. **Robustness Gap**: The difference between the accuracy of the model on clean images and the accuracy of the model on clean images after applying the defense, such as adversarial training.\n",
    "\n",
    "5. **Similarity**: The similarity between the original and adversarial examples. Supported metrics are SSIM and PSNR.\n",
    "\n",
    "6. **Perturbation Efficiency**: The effectiveness score is the attack success rate divided by the perturbation distance. The higher the score, the more effective the attack.\n",
    "\n",
    "There are two ways to adversarially evaluate the model using `advsecurenet` API. We can either utilize the `Attacker` or we can manually iterate over the dataset and calculate the evaluation metrics.\n",
    "\n",
    "**Note:** `advsecurenet` CLI also supports evaluation. You can further use Distributed Evaluation to evaluate the model on multiple GPUs using the CLI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adversarial Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the Attacker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from advsecurenet.attacks.gradient_based import FGSM, LOTS\n",
    "from advsecurenet.shared.types.configs.attack_configs import (\n",
    "    FgsmAttackConfig,\n",
    "    LotsAttackConfig,\n",
    ")\n",
    "from advsecurenet.shared.types.configs.attack_configs.attacker_config import (\n",
    "    AttackerConfig,\n",
    ")\n",
    "from advsecurenet.models.model_factory import ModelFactory\n",
    "from advsecurenet.datasets.dataset_factory import DatasetFactory\n",
    "from advsecurenet.attacks.attacker import Attacker\n",
    "from advsecurenet.dataloader.data_loader_factory import DataLoaderFactory\n",
    "from advsecurenet.shared.types.configs.preprocess_config import (\n",
    "    PreprocessConfig,\n",
    "    PreprocessStep,\n",
    ")\n",
    "from advsecurenet.shared.types.configs.device_config import DeviceConfig\n",
    "from advsecurenet.utils.adversarial_target_generator import AdversarialTargetGenerator\n",
    "from advsecurenet.datasets.targeted_adv_dataset import AdversarialDataset\n",
    "from advsecurenet.shared.adversarial_evaluators import adversarial_evaluators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the model\n",
    "model = ModelFactory.create_model(\n",
    "    model_name=\"resnet18\", num_classes=10, pretrained=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lets define the preprocessing configuration we want to use\n",
    "preprocess_config = PreprocessConfig(\n",
    "    steps=[\n",
    "        PreprocessStep(name=\"Resize\", params={\"size\": 32}),\n",
    "        PreprocessStep(name=\"CenterCrop\", params={\"size\": 32}),\n",
    "        PreprocessStep(name=\"ToTensor\"),\n",
    "        PreprocessStep(\n",
    "            name=\"ToDtype\", params={\"dtype\": \"torch.float32\", \"scale\": True}\n",
    "        ),\n",
    "        PreprocessStep(\n",
    "            name=\"Normalize\",\n",
    "            params={\"mean\": [0.485, 0.456, 0.406], \"std\": [0.229, 0.224, 0.225]},\n",
    "        ),\n",
    "    ]\n",
    ")\n",
    "\n",
    "# Define the dataset\n",
    "dataset = DatasetFactory.create_dataset(\n",
    "    dataset_type=\"cifar10\", preprocess_config=preprocess_config, return_loaded=False\n",
    ")\n",
    "test_data = dataset.load_dataset(train=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the dataloder\n",
    "dataloader = DataLoaderFactory.create_dataloader(dataset=test_data, batch_size=32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the device config\n",
    "device = DeviceConfig(processor=\"mps\")\n",
    "\n",
    "# Define the fgsm config\n",
    "fgsm_config = FgsmAttackConfig(\n",
    "    targeted=False,\n",
    "    epsilon=0.1,\n",
    "    device=device,\n",
    ")\n",
    "\n",
    "# Now we can define the attack\n",
    "attack = FGSM(config=fgsm_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# lets check the evaluators\n",
    "print(adversarial_evaluators.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluators = [\n",
    "    \"attack_success_rate\",\n",
    "    \"robustness_gap\",\n",
    "    \"perturbation_distance\",\n",
    "    \"similarity\",\n",
    "]\n",
    "attacker_config = AttackerConfig(\n",
    "    model=model,\n",
    "    attack=attack,\n",
    "    dataloader=dataloader,\n",
    "    device=device,\n",
    "    return_adversarial_images=False,\n",
    "    evaluators=evaluators,\n",
    ")\n",
    "\n",
    "attacker = Attacker(config=attacker_config)\n",
    "\n",
    "attacker.execute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can also evaluate the transferability of the adversarial examples\n",
    "\n",
    "# define a target model\n",
    "vgg16 = ModelFactory.create_model(model_name=\"vgg16\", num_classes=10, pretrained=False)\n",
    "\n",
    "evaluators = [\"transferability\"]\n",
    "attacker_config = AttackerConfig(\n",
    "    model=model,\n",
    "    attack=attack,\n",
    "    dataloader=dataloader,\n",
    "    device=device,\n",
    "    return_adversarial_images=False,\n",
    "    evaluators=evaluators,\n",
    ")\n",
    "\n",
    "attacker = Attacker(config=attacker_config, target_models=[vgg16])\n",
    "\n",
    "attacker.execute()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the Evaluator\n",
    "\n",
    "If you don't want to use the `Attacker` class, you can manually iterate over the dataset and calculate the evaluation metrics. Here is an example of how to do that:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from advsecurenet.evaluation.adversarial_evaluator import AdversarialEvaluator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with AdversarialEvaluator(evaluators=[\"attack_success_rate\"]) as evaluator:\n",
    "    for images, labels in dataloader:\n",
    "        model.eval()\n",
    "        model = model.to(device.processor)\n",
    "        images = images.to(device.processor)\n",
    "        labels = labels.to(device.processor)\n",
    "        adv_images = attack.attack(model, images, labels)\n",
    "        evaluator.update(\n",
    "            model=model,\n",
    "            original_images=images,\n",
    "            true_labels=labels,\n",
    "            adversarial_images=adv_images,\n",
    "            is_targeted=False,\n",
    "        )\n",
    "\n",
    "print(evaluator.get_results())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benign Evaluation\n",
    "\n",
    "You can also evaluate the model on clean images using the `Tester` class. Here is an example of how to do that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from advsecurenet.shared.types.configs.test_config import TestConfig\n",
    "from advsecurenet.evaluation.tester import Tester"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_config = TestConfig(\n",
    "    model=model,\n",
    "    test_loader=dataloader,\n",
    "    processor=device.processor,\n",
    "    topk=3,  # You can change this to any value you want\n",
    ")\n",
    "\n",
    "tester = Tester(config=test_config)\n",
    "\n",
    "tester.test()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "adv2",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
